Synthetic data generation method for data-free knowledge distillation in regression neural networks

Knowledge distillation is the technique of compressing a larger neural network, known as teacher, into smaller student, while still trying to maintain performance network much possible. Existing methods knowledge are mostly applicable for classification tasks. Many them also require access data used train teacher model. To address problem regression tasks under absence original training data, previous work has proposed data-free method where synthetic generated using generator model trained adversarially against student These and their labels predicted by then In this study, we investigate behavior various generation propose new strategy that directly optimizes large but bounded difference between Our results on benchmark case study experiments demonstrate allows learn better emulate more closely.